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To optimize the bulk-insulating behavior in the topological insulator materials having 
the tetradymite structure, we have synthesized and characterized single-crystal samples of 
Bi2_ a; Sb a: Te3_ySey (BSTS) solid solution at various compositions. We have elucidated that there 
are a series of "intrinsic" compositions where the acceptors and donors compensate each other and 
present a maximally bulk-insulating behavior. At such compositions, the resistivity can become as 
large as several flcm at low temperature and one can infer the role of the surface-transport channel 
in the non-linear Hall effect. In particular, the composition of Bi1.5Sbo.5Te1.7Se1. 3 achieves the 
lowest bulk carrier density and appears to be best suited for surface transport studies. 

PACS numbers: 73.25.-K, 74.62.Dh, 72.20.My, 73.20.At 



I. INTRODUCTION 

The three-dimensional (3D) topological insulator 
(TI) 1-5 possesses spin- momentum- locked gapless surface 
states that are expected to host a variety of interest- 
ing quantum phenomena*^ While a number of materi- 
als have been identified to be 3D TIspT— most of the 
known TI materials arc poor insulators in the bulk, and 
finding ways to achieve a bulk-insulating state is an im- 
portant current theme.— ~— Recently, a large bulk re- 
sistivity together with clear quantum oscillations from 
the surface state was observed in a ternary tetradymite 
TI material Bi 2 Te 2 Se (BTS)^^ In this material, the 
chalcogens (Te and Se) are separated into different lat- 
tice sites, forming the ordered Te-Bi-Se-Bi-Te quintuple 
layers. This ordering provides the chemical environment 
suitable for reducing defect formations. 26 As a result, the 
chalcogen-ordered tetradymite TI is a promising material 
for approaching the intrinsic topological-insulator regime 
where the bulk carriers are negligible. Since the residual 
bulk carrier density in BTS was found to be of the order 
of 10 17 cm~ 3 , 26 it is desirable to find an appropriated 
route to further improve the bulk-insulating properties 
of the chalcogen-ordered tetradymite TI material. 

In this respect, the tetradymite solid solution 
Bi 2 _ :c Sb a: Te3_ y Sej / (BSTS) is interesting, because it has 
been known that some special combinations of x and y in 
this solid-solution system yield very low conductivity.— 
Except for a narrow composition region close to Sb 2 Se3, 
BSTS takes the same rhombohedral structure as its three 
end members Bi 2 Te3, Bi 2 Se3, and Sb 2 Te3, all are known 
to be TIsjii~— Hence, BSTS is naturally expected to be 
a TI as long as its structure is rhombohedral. Note that 
the BTS compound, where a perfect chalcogen ordering 
is presumably achieved, can be regarded as a member of 
this family [x = 0, y = 1). 

Originally, the BSTS solid solution attracted atten- 
tion because of its promising thermoelectric properties 
for near room-temperature applications^ Decades of ef- 
forts were devoted to optimizing the thermoelectric per- 
formance through tuning the composition and/or doping 



in the degenerate regime, resulting in a dimensionless fig- 
ure of merit ZT (= S 2 T / pn, where S is the thermopower, 
p is the electrical resistivity, and n is the thermal con- 
ductivity) close to 1. By contrast, nearly no effort was 
made to obtain insulating behavior in this family; never- 
theless, because two prevailing types of charged defects 
that exist in BSTS [(Bi,Sb)/Te anti-site defects and the 
Se vacancy defects] introduce carriers of opposite signs, 
a careful tuning of x and y would, in principle, substan- 
tially cancel the bulk carriers and allow one to maximize 
the bulk insulating properties. In fact, based on a sys- 
tematic study of polycrystalline BSTS samples at room 
temperature, Teramoto et al. inferred 2 - that a series of 
such "intrinsic" compositions exist, and that the values 
of x and y for such compositions are linearly coupled. 
Unfortunately, the temperature dependence of the resis- 
tivity was not measured in their experiment, so it was 
not clear to what extent an insulating state was actually 
realized. 

In the present work, we performed a systematic study 
of the transport properties of the BSTS solid solution for 
a wide range of compositions and temperature, with the 
aim of maximizing the insulating property. It was found 
that the compositions suggested by Teramoto et al. do 
not exactly correspond to the optimal compositions for 
insulating behavior. Instead, we determined a series of 
optimized compositions, where the relationship between 
x and y is obviously non-linear. The BSTS crystals at 
the optimized compositions with < x < 1 were found 
to present a large bulk resistivity exceeding 1 ficm at low 
temperature, and their transport properties at high tem- 
perature signify the existence of a small activation gap. 
This result gives evidence for the existence of a series 
of BSTS solid solutions where the electron and hole car- 
riers are nearly compensated. In particular, it appears 
that at (x, y) — (0.5,1.3) (Bir.5Sbo.5Te1.7Se1. 3) the com- 
pensation is maximally realized. Although the chances 
of observing the Shubnikov-de Haas (SdH) oscillations in 
BSTS samples are not very high, one can use the non- 
linear Hall effect which is always observed at low temper- 
ature as a tool to infer the role of the surface-transport 
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channel. 



II. EXPERIMENTAL DETAILS 

The single crystals of Bi2- a ,Sb a: Te3_ySe !/ were grown 
by melting stoichiometric amounts of high purity ele- 
ments [Bi, Sb, and Te were 6N (99.9999%), while Se was 
5N (99.999%)] in sealed evacuated quartz tubes at 850 °C 
for 48 h with intermittent shaking to ensure a homogene- 
ity of the melt, followed by cooling slowly to 550 °C and 
then annealing at that temperature for 4 d. The result- 
ing crystals have a typical domain size of up to several 
centimeters and are easily cleaved along the (111) plane 
to reveal a shiny surface. The crystal structure of each 
sample was investigated by the x-ray diffraction (XRD) 
analysis with the Cu Ka emission, which was performed 
on powders obtained by crushing the crystals. Lattice 
parameters were refined by a least-squares fitting to the 
XRD data with the consideration of the zero shifts. All 
the samples used for transport measurements were con- 
firmed to be single domain by using the x-ray Laue anal- 
ysis, and they were cut into thin bar-shaped specimens 
with the typical thickness of 100 /im. The in-plane resis- 
tivity p xx and the Hall resistivity p yx were measured by 
using the standard six-probe method in a Quantum De- 
sign Physical Properties Measurement System (PPMS-9) 
down to 1.8 K, shortly after the electrical contacts were 
prepared by using room-temperature-cured silver paste. 



III. INTRINSIC COMPOSITIONS 

Figures l(a)-(c) present how the compositions of the 
solid solution were optimized for the insulating behavior. 
Initially, we synthesized and characterized samples for 
the "intrinsic" compositions suggested in Ref. |28| How- 
ever, it turned out that, except for the (x, y) — (0.5, 1.3) 
case, all those samples were only poorly insulating or 
even metallic, with a nearly T- independent Hall coeffi- 
cient signifying the existence of electron carriers with the 
density of 10 18 -10 19 cm -3 , which indicated that there 
are too much Se vacancies that are not sufficiently com- 
pensated by the (Bi,Sb)/Te anti-site defects. Therefore, 
we tried to optimize the composition by increasing the 
Se content y while fixing the Sb content x. The exam- 
ples for x = and 1 are shown in Figs. 1(a) and (b). We 
have also performed similar optimization along the struc- 
tural phase boundary [line A-B in Fig. 1(d)], in which 
one must simultaneously vary the values of x and y. As 
shown in Fig. 1 (c), the composition (x, y) = (1.5,2.4) 
suggested in Ref. l28| yielded only metallic samples; mov- 
ing to (x,y) — (1.4,2.44) was not sufficient, and finally 
at (x, y) — (1.25, 2.5) we found an insulating behavior. 

The summary of the optimized compositions are shown 
in the composition-structure diagram [Fig. 1(d)], to- 
gether with the linear "intrinsic" composition line sug- 
gested in Ref. l28l . The possible errors in our experiment, 
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FIG. 1: (Color online) (a,b) Temperature dependences of p xx 
for BSTS samples with fixed x and different y, showing how 
the compositions were optimized for the insulating behavior; 
panel (a) is for x = (BhTe3- y Se y ) and panel (b) is for x 
= 1 (BiSbTea-ySey). (c) p X x{T) data used for optimizing 
the composition along the structural phase boundary [line A- 
B in panel (d)]. (d) Composition-structure diagram of the 
Bi2- a! Sba ; Te3- a Se^ system. The circles denote the "intrin- 
sic" compositions determined in the present work, while the 
dashed line denotes those suggested in Ref. |28j . This dia- 
gram was determined by first fixing the Sb content x and 
optimizing the Se content y for that x; the possible error in 
the optimized y value for a given x is the step size in the 
optimization process and is represented by the symbol size. 

which mostly come from the finite step size in the op- 
timization process, are represented by the symbol size. 
Obviously, the relationship between x and y in the true 
"intrinsic" compositions is not linear but shows an up- 
ward curvature in this diagram. It should be noted that 
Teramoto et al. used polycrystalline samples and only 
measured the conductivity at room temperature^ it is 
therefore not surprising that the present result does not 
agree with their conclusion. 

IV. CRYSTAL STRUCTURE 

Figure 2(a) shows the powder XRD patterns for the 
series of BSTS solid solutions at the optimized composi- 
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TABLE I: Important parameters obtained for the opitimized compositions. A and A* are the thermal activation energies 
determined from the temperature de pendenc es of p xx and _Rh, respectively. The effective acceptor concentration is 
estimated from the relation R' 1 « e^/NfN v exp(-A*/fe B T), where N v = 5.2xl0 18 cm -3 (Ref. [H) was assumed. Note that, 
among the samples with the same composition, A and A* vary by ~10% and could vary by a factor of three, probably 
due to different levels of defect concentrations. 
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plot [Fig. 2(b)]. This indicates that the ordering of 
the chalcogen layers is preserved to some extent despite 
a large change in stoichiometry. To understand this 
observation, it is useful to recall the structure of the 
tetradymite, which is composed of stacked A^B^ 1 (A v 
= Bi, Sb; B VI = Te, Se) quintuple layers. Within the 
quintuple layer, the atoms are arranged in the sequence 
of B VI (1)-A V -B VI (2)-A V -B VI (1), where B VI (1) and 
B VI (2) are two inequivalent crystallographic sites. Be- 
cause Se has a larger electronegativity than Te, the Se 
atoms preferentially occupy the B VI (2) site and the re- 
maining (y — l)Se + (3 — y)Te atoms are distributed ran- 
domly in the B VI (1) site when y is larger than 1^2 Such 
an asymmetric arrangement of the chalcogen layers bears 
similarity to that in BTS and accounts for the XRD re- 
sults. 
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FIG. 2: (Color online) (a) Power XRD patterns for the series 
of BSTS samples at optimized compositions. The pattern for 
Bi2Te3 is also shown for comparison. All the diffraction peaks 
can be indexed based on the rhombohedral structure with the 
R3m space group, (b) Enlarged view of the data near the 
(108) peak. Arrows mark the (107) and (0012) peaks related 
to the ordering of the chalcogen layers. 

tions, together with the Bi2Te3 data which is shown for 
comparison. The patterns for all the samples are essen- 
tially the same and can be well indexed with the rhom- 
bohedral structure (space group R3m), confirming that 
the solid solutions maintain the same crystal structure as 
their binary end members Bi2Te3, Bi2Se3, and Sb2Te3. 
The refined lattice parameters are summarized in Table 
I. As the system moves toward Sb2Se3, both the a and 
c lattice parameters tend to decease, which is reasonable 
because the ionic radius of Sb (Se) is smaller that that 
of Bi (Te). 

It is important to note that the (107) and (0012) 
peaks, both of which are characteristic of the compounds 
with the BTS-type chalcogen ordering, are present in 
all the BSTS samples, as can seen in the magnified 



V. RESISTIVITY 

Figure 3(a) shows the typical temperature dependences 
of Pxx for the optimized compositions. It should be noted 
that the p xx value was found to be sample dependent 
and could vary by a factor of three within the same com- 
position, probably due to different levels of defect con- 
centrations. Nevertheless, the activated behavior was 
essentially reproducible within the same batch and the 
variation in terms of the activation energy was about 
10%. The magnitude of p xx at room temperature in 
Fig. 3(a) ranges from 25 to 140 mOcm, which is gen- 
erally larger than that found in BTSi^S Except for the 
(x,y) = (1.25,2.5) composition which has the largest Se 
content, the overall temperature dependence of p xx is 
similar and can be divided into three regimes: 

(i) Activated regime: In the temperature range above 
^100 K, the p xx {T) data can be fitted with the Arrhenius 
law 

p xx ~ exp(A/fc B T), (1) 

where A is the activation energy and &b is the Boltzmann 
constant. In Fig. 3(b), we show the Arrhenius plot of 
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FIG. 3: (Color online) (a) Temperature dependences of p xx 
for the series of BSTS samples at optimized compositions. 
Note that the vertical axis is in the logarithmic scale with 
base 10. (b) and (c) are the Arrhenius plot \[og 10 p xx vs 1/T] 
and the 3D-VRH plot [log 10 p xx vs T~ 1/4 } of the p xx (T) data, 
respectively. Dashed lines represent linear fittings to the re- 
gions where the activation and the VRH behaviors can be 
recognized. 

the data together with linear fittings to obtain A. The 
A values are summarized in Table I, and they vary from 
22 to 53 meV; however, since the temperature range of 
this activated behavior is not very wide, the obtained A 
should not be taken too literally. 

(ii) Variable-range hopping regime: Below the temper- 
ature range of the activated behavior mentioned above, 
the p xx (T) appears to be better described by the 3D 
variable-range hopping (VRH) behavior, 

p xx ^eM(T/T )-^% (2) 

where To is a constant which depends on the density of 
states at the Fermi level Ep. For example, in the data for 
Bi2Te1.95Se1.05 (marked by square in Fig. 3), the VRH 
behavior holds for 60 - 140 K, while the activated behav- 
ior holds for 110 - 300 K. However, it should be noted 
that the temperature range where this VRH behavior 
appears to hold can have a significant overlap with the 



activated temperature range. For example, in the case 
of Bii.25Sbo.75Tei.3Sei.7 (marked by diamond in Fig. 3) 
where the distinction is the most ambiguous, the activa- 
tion behavior and the VRH behavior appear to hold in 
similar ranges, 85 - 140 K and 90 - 170 K, respectively. 
Hence, the distinction between the two transport mech- 
anisms can be a subtle issue, and it probably depends on 
the level of disorder in the sample. 

(hi) Saturation regime: At low temperature, p xx tends 
to saturate, rather than to diverge as expected for in- 
trinsic semiconductors, implying the existence of some 
extended states at E-p in the zero-temperature limit. In 
this regime, the magnetic-field dependence of the Hall re- 
sistivity p yx is generally non-linear (as will be described 
in detail later), which points to the existence of two 
or more transport channels. In the BTS compound, it 
was elucidated^ that the topological surface state and 
a degenerate bulk impurity band both contribute to the 
low-temperature saturated resistivity. In addition, our 
recent study of the Bi1.5Sbo.5Te1.7Se1. 3 compound has 
identified^ the third contribution of the accumulation 
layer (which is caused by the surface band bending and 
is topologically trivial) in samples where the Hall coeffi- 
cient is positive at low temperature. 



VI. HALL COEFFICIENT 

To further characterize the transport properties, we 
have performed the Hall measurements. Figure 4(a) 
shows the temperature dependences of the low-field Hall 
coefficient i?H, defined as i?H = Py X /B near B = 0, 
for the same series of samples. Above ~150 K, p yx is 
always linear in B so that p yx /B is magnetic-field in- 
dependent, and the resulting Rn is positive and shows 
a thermally-activation behavior in all the BSTS sam- 
ples except Bio.75Sbi.25Teo.gSe2.5. The positive and ac- 
tivated Rn(T) behavior indicates that the dominant 
charge carriers are thermally-excited holes in the bulk 
valence band. By contrast, the high-temperature i?n in 
Bio.75Sbi.25Teo.5Se2.5 is negative and only weakly depen- 
dent on temperature, which indicates that the electron 
carriers created by Se vacancies in this compound are de- 
generate near room temperature. Thus, the sign change 
in the high-temperature i?H at the largest Se content y 
in our series of samples reflects a change in the domi- 
nant defects from the Bi(Sb)/Te anti-sites (acceptors) to 
the Se vacancies (donors). Note that in those samples 
both acceptors and donors are expected to exist, and the 
observed charge-carrier concentration is likely to be de- 
termined by their competition. 

From the Arrhenius plot of Rn(T) [Fig. 4(b)], one ob- 
tains the effective acceptor concentration Nj~ together 
with the effective activation energy A*, both are sum- 
marized in Table I. It should be noted that, among the 
samples with the same composition, the variation in A* 
was within ~10%, while could vary up to a factor of 
three, reflecting the variation in the resistivity behavior 
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the moment. 

At lower temperature below 100 K, p yx (B) is no longer 
a linear function of B and i?H becomes very much depen- 
dent on composition. Figure 4(c) shows this variation in 
terms of the Rr value at 1.8 K plotted vs y.— The largest 
was observed in Bi1.5Sbo.5Te1.7Se1. 3, where i?H = 
—910 cm 3 /C at 1.8 K. This value would naively corre- 
spond to a very low carrier density of 7x 10 15 cm -3 ; how- 
ever, as we have demonstrated for BTS,— such a naive 
estimate based on the low-field i?n is not reliable when 
there are multiple transport channels that cause p yx (B) 
to become non-linear. Indeed, p yx (B) was found to be 
non-linear in Bi1.5Sbo.5Te1.7Se1. 3 [Fig. 4(d)], and an ac- 
curate estimate of the bulk and surface carrier densities is 
difficult without employing additional information from 
SdH oscillationsi 26 i 33 Nevertheless, by using the simple 
two-band model to consider both the bulk and surface 
transport channels 26 (note that the accumulation layer 31 
is irrelevant here since the slope of p yx (B) is negative), 
one can make a reasonable fitting to the data [solid line in 
Fig. 4(d)], which yields the bulk carrier density 1.8 xlO 16 
cm~ 3 , the bulk mobility 73 cm 2 /Vs, the surface carrier 
density 1.5xlO n cm -2 , and the surface mobility 2900 
cm 2 /Vs. (In the fitting procedure, the parameters were 
constrained by the measured p xx value in T.) Based 
on this fitting result one can estimate the contribution 
of the surface transport to the total conductance to be 
~5%, which is reasonably large considering the thickness 
(120 pm) of the measured sample. 



VII. DISCUSSIONS 



FIG. 4: (Color online) (a) Temperature dependences of the 
low-field Hall coefficient Rn for the series of BSTS samples 
at optimized compositions. Inset shows an enlarged view 
of the data between 150 and 300 K. (b) Arrhenius plot of 
Rh{T) at high temperature for the series of samples except 
Bio.75Sbi.25Teo.5Se2.5 where the activation behavior was not 
clearly observed; the data are shifted vertically for clarity. 
Dashed lines are the Arrhenius-law fittings to extract the ac- 
tivation energy A*, (c) The value of Ru at 1.8 K plotted as a 
function of the Se content y. (d) Magnetic-field dependence of 
Pyx for the Bi1.5Sbo.5Te1.7Se1. 3 sample at 1.8 K; solid line is 
a two-band-model fitting (Ref. 2g) to the data. The obtained 
fitting parameters give the surface conductance of 1.4xl0 -4 
Q -1 and the bulk conductivity of 0.21 S7 _1 cm -1 ; since the 
measured sample was 120 pm thick, the surface contribution 
was 5% of the total conductance. 



within the same batch mentioned in Sec. V. One can see 
that the A* values are not much different from the A val- 
ues derived from the p xx (T) data for the same samples, 
and the difference is less than 30%. This small differences 
can probably be ascribed to the temperature dependence 
of the mobility. Intriguingly, and A* seem to be cor- 
related; namely, larger is accompanied with larger 
A*. However, the origin of this behavior is not clear at 



For the transport studies of the topological surface 
states, it is desirable that their SdH oscillations are ob- 
served. In the present series of BSTS samples, we did 
not observe clear SdH oscillations below 9 T. Most likely, 
whether the SdH oscillations can be observed depends 
crucially on the homogeneity and the value of the surface 
carrier density n s ^ which may vary between samples 
and, even in the same single crystal, between different 
cleaves. The chances of observing the SdH oscillations 
were roughly 10% in our BSTS samples. On such lucky 
occasions, we were able to see both Dirac holes and elec- 
trons on the surface of Bi1.5SboKTe1.7Se1. 3 samples, and 
the result was reported in Ref. |3l|. It should be empha- 
sized, however, that even though the SdH oscillations are 
not always observed, one can achieve a surface-dominated 
transport by making sufficiently thin samples of BSTS. 31 

As we have already discussed in Sec. VI, the low-field 
i?H value in BSTS does not simply reflect the bulk carrier 
density, but it is also influenced by the contribution of the 
surface carriers which presumably have a higher mobility 
than the bulk carriers. In view of this situation, the non- 
trivial composition dependence of Ru observed at 1.8 K 
[Fig. 4(c)] can be interpreted in two ways: One possibil- 
ity is that it may signify the sign change of the surface 
carriers as the Se concentration y is increased. This is 



6 



possible because the surface band structure of BTS is 
similar— to that of Bi 2 Te3 where the Dirac point is lo- 
cated below the top of the valence band,— whereas with 
increasing y the surface band structure is expected to ap- 
proach that of Bi2Se3 where the Dirac point is isolated 
in the bulk gap;' 1 - therefore, if the chemical potential is 
pinned to the impurity band located just above the bulk 
valence band^ the nature of the surface carriers could 
change from Dirac electrons to Dirac holes. The other 
possibility is that the change in i?H is a reflection of the 
change in the nature of the degenerate bulk carriers in 
the impurity band. This is also possible because at low 
y the chemical potential is likely to be located below the 
center of the acceptor impurity band, whereas with in- 
creasing y the increasing amount of donors (due to Sc 
vacancies) would cause the acceptor impurity band to be 
gradually filled and move the chemical potential to above 
the center of the impurity band. To further address this 
issue, systematic studies employing the SdH oscillations 
would be necessary. 



a maximally bulk-insulating state. The powder XRD 
patterns suggest that at those compositions optimized 
for the insulating behavior, the crystal structure has an 
ordering of the chalcogen layers in which the Sc atoms 
preferentially occupy the middle of the quintuple layer 
and the remaining Se and Te atoms randomly occupy 
the outer-most layers. Except for Bio.75Sbi.25Teo.sSe2.5 
which is close to the structural instability, all the 
BSTS samples at the optimized compositions show 
large p xx values exceeding 1 J7cm at low temperature, 
together with an activated behavior at high temperature 
signifying the existence of an activation gap for the bulk 
carriers. The B dependence of p yx at low temperature 
points to the role of the surface transport channel, 
and the non-trivial composition dependence of the 
low-field i?H reflects either the change in the surface 
band structure or the change in the bulk carriers in the 
impurity band. The \Rb\ value at 1.8 K was found to be 
the largest in Bi1.5Sbo.5Te1.7Se1. 3 and this composition 
appears to have achieved the lowest bulk carrier density. 



VIII. CONCLUSION 

In this work, we have elucidated in the 
Bi 2 -^Sb I Te3_ y Sej / (BSTS) solid solution the existence 
of the "intrinsic" compositions where the acceptors due 
to the (Bi,Sb)/Te anti-site defects and the donors due to 
the Se vacancy defects compensate each other and realize 
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